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The Bural ZSncatioil Progran (EEP) of thd^ Horthvest 
Eegio&al Edocationalw^^^Laboxatcry (SliHEL) is^ developing a' strategy for 
involving^ rural ccaiaanities asd school systems in a -systematic 
problem solving -process.. This 5rfral Futures Developacent (BrB) 
Sfcrategy is b^sed on the theoretical vorxs of such people as 
Hayelbck# lippir, Bales^ 'rfilliaHSon^ Schauck and Sutkel and* on 
Yii^BL's lieis based develcpac^nt efforts* Shis -paper presents^ (1) an 

'overview of i:he E?D Strategy and its related productSj. (2) a 

♦discussion of the three phases of the 'evaluation of the*strategy and 
(3) an examination of^the interactive procedures between developers 
and evaluators ia the 'f ornative evaluation of products which are 

"integral to strategy installation^. Essentially, the strategy consists 
of (1) the tra'ining and field involvesaent of process facilitators . 
(outside cfiange agents) .rfho intro.dnce the' strategy and share problea 
solving and" decisi*on zraking skills w^th lodal peoplej. (2) the 
deveiopiaent^of a representative local school-coaaunity • group^ and (3) 

♦the collaborative invclvesent of the school sysj:ea and 
' school -co3Baani^y group in identifying and solving local educational 
pf obi ens. (a^>^cryHC), ' . ^ ^ ' ' : 
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CEVELGPERS AND iEVALUATORS ENGAGED IM 
FORMATIX^ PRODUCT CEVELePMEl^T 



Abstract . * : 

- • This paper presents (1) ^ overview of a strate^ for Involving 
rural cen^munilies and schoo! systems in solving local educational problems^ 
• 12) a discussion of the evaluation phases cf the strategy^ and (3i an 
eAairfnaticn of ti^e interactive procedures bet»veen developers and evakjetoi^ 
In the forfnatlve evaluation of products integral to the strategy. Development 
cZ: J evaluation roles are discussed in relatitJn to the exploratoty test . 
proc^ures for identiiying and selecting reviewers, developing test 
schedules, constructing ins trun:ents, conducting thelest^ and fjeporting 
^e results. * ^ . 

Introduction ' . • . ' 



The Rural Hducation Program {REPJ of the Northyvest Regional 
Educdti^na! Laboratory (l^WREL) is developing a strategy for involving 
rural cx^mhnjnili^s and school systems in a systematic probfem solving 
process, T-his Rur^i Futures Development XRFD) Strategy iS based on 
the theoretical works of such people as Havelock^ Lippit, Bales, V/illiamson, 
Schmuck end Runkel and dn our ov/n field based development efforts. 
This^paper presents. (1) ^n overview of the RFD Strategy and its related . ^ 
products, (2) e discussion of the three phases of the evaluation of the 
strategy and (3) an examination of the interactive prpcedures b^fween 
developers and evaiuators' in the formative evaluation of products which 
are integral to strategy installation.' *' , ^ ^ 

An Overview of the RFD Strategy ./. , • - 

Essentially, the strategy consists of fl) the training and field ' 
involvement of process f^dcilitators (outside change agents) who introduce 
the strategy and share problem solving and decision making skills v/ith 
local people', (2) Oie development oT a representative local school -community 
group, and (3) the collaborative involvement of the sdiool system and 
school-community group in identifying and solving local educational 
problems. . ^ * ' ^ . t * 



AM OV£R?^aV OF EVALUATiOM CF THE STRATEGY 

During the pas* fevi? years, the prcduct and process pieces of the 
^RFD Strategy have been in various stages of developmeht. Tl^e products ^ 
currency reflect varying degrees of reliance on theoretical concepts and 
on field bas^ development and testing. Since ihe sumnier of. 1974 the 
primary thrust of the Progr^ has been on the installation and, evaluation'^ 
of an integrated RFD Strategy. ^To this ertd the Program 3oegBn the first 
trial of the int^rated strategy in SQ^tember 197**, in^what is designatecf 
as Sate ^. Currently a second instaHaticn— designated as Site B%-is under- 
way^* - * - *- ^ * 

' Site A. Site A serves as the setting for the first tryout of^e 
integrated JIFO Strategy. !n Ihis'site, evaluators are chiefly concerijed 
v/ith providing data to facilitate the refinement and delia^eation of the 
RFD Strategy definition prior to its inslaSIation in Site 8. 

Site 8 .^ Site B 'involves a test of the importance^ efficacy^ and 
integration of the RFD Strategy and related KFD proddcts. In Site 
evaluators are concerned with (l).the <Jegree to which the Strategy was 
implemented (as reflected in its process objectives) and (2) the extent 
to v/hich the Strategy's outc^e objectives are met. Unanticipated conse- 
quences of RFD adoption v^cH^salso be*noted- ^ - - . 

The exploratory tests . The exploratory /tests provide the development 
stafr with information that rf^ey can use in revising specific products 
preparatory to their use v/hen the integrated strategy is installed in a 
site. . - * * ^ ' 

- Thus^ Uie strategy evaluation includes three major phases: (1) 
Jhe exploratory* test ofiindividual products^ (2) the evaluation of the fir^t 
installation of the strategy v/hich is intended to identify the.gaps^^ redun^ 
dancies and major v/eaknesses^ in the strategy leading to its refinement 
and' (3) the evaluation of a second installation which is to be comparative ^ 
and thus a more rigorous examination of the effects of the RFD Strategy . 
The paper will focus on the inter-action betv/een developers and evaluators 
in* the exploratory test of products within the context of ^the overalf evaFua- 
tion of the Strategy- 



Th.e foncwing products^- developed for use v/ilh^ihe strategy; 
were included ii? th^ expJoratofY teste. Under eadfi prodisct Sitie a ^ 
short description is provided or the contents ^d purpQse cf the product. 

1- RFD ?>!anual for^d:oo1-Cc^:^:u^^^/ Process Facilitators 



The process^ facilitator manual is designed to help teams 
oj process facilitators (PF teams] v/ork with the school 
staff* the school boarjd^ sdiodi J^ders^ and the school- 
community group <SCG} in the local School-Community 
Proccess of the RFD Strategy- its primary purpose is 
to provide these teaihs with a" structure for their consulta- 
tion and a- set of procedures and resources for adapting 
the process to meet individual school and cx)Oimunity needs- 
A secondary^purpose is to Jhelp process facilitators to main- 
tain positive team relationships* as welLas to strengfiiea 
th^r relationship to their hiring agency. 

RFD Guide for Training School-Community Process Facilitators 



This training guide is designed to enable trainers to provide 
appropriate traixjing* activities fo^^rocess facilitators.. 
Such graining v/ill enable them to gain and exhibit a wide- 
'.ranjg^ of skills essentia! to their- work'. - * ^ 

RFD Guide for Schools - * * 



The school guide is directed at school staff members and is 
designed^to suppoiH and guide their participation. The guide 
contain^ background infoVm^tion, step-by-step gufdelines^ 
arid resources. • ' 

' ■ ; ■ ■ ■ ■ 

" i\. RFD Notebook for School-Community Groups- 



TheSCG notebook isSdirected at SCG representatives from 
the school staff, the sti^ent* body, and various community ^ 
organizations. It, too, contains materials that are designed 
to'^suRport and" guide the group'' s' particf pa tio/i. 



* 5. RFD Guide for School- Svards 



The school board guide is dssigced to provide school board 
*" members witi\jr-ateria!s and procedures for recording^ organi- 
zing^ and retrieving infonnaUon necessary for their participa- 
^on in ^e RFD SU^ategy. In the guide a systematic approach 
- , ' tojprobJem solving is also ouHih^^ afong tvith .a discussion 
. of the leadership styles and Unctions that zre compatible 
v.ith Lhe RFD Strat^y. \ - 

Evaluative Questions , ' i^ * ♦ • 

J The explora&ry tests are formative product evaluations designed 

to , provide develop'ers with valid and reiiafaie data for use in product 
r^visfiefn- ^4^ese tests focused ^n questions, ijientified by developers, 
in* regard to the product's objectives, content, format^ and intended 
use. The major questions* for the exploratory tests are as follov/s: 

A- Goals and Objectives ^ \ ^ 

I . * Are the goals and objectives of^h^ product appropriate 

and v/ell defined? * - , J * ^ r - 

2- ^ Are there goals-and objectives which should fae addec}/ 
^ eliminated*, or revised? ^ ^ / 

* 8. Content of the Product ^ 

Does the product content match with the specified goals 
and objectives? * - * . 

^ / 4. Is the content of. sufficient scope and detail to serve 
/ the needs x)f intended u§ers? . 

* 5. . Are the content and writiog 'style appropriate for the 
prospective audiences? ^' * 

* 6. 'Should any material be added to or deleted from the - . 
' ^ product content? ^ ' * _ ' 

7., Are^.there any c^'tical competitors for the product? 
/ 8. Is the content compatible v/ith the RFD Strategy? 
C. Intended. Uses of the Product 

9. Does the product consider the importan^rch^acteristics 

of its prospectjve-a^udjences? ^ \ * 

Id. Is this product piece esLsentiiFta the' RFD Strategy? 
11. Are the reference and resource materials appropriate* 
and a"dequate to meet the needs of intended users of • 
^ the prdtiuct? ' , ' - . 



D- Fonnat and Graphics - ^- 

12. is the. matsrial sequen^d appropriately? , 

13. - Does the table of contents provide ease of .reference ^ 
. to the product? .... — ^ 

Vi. Is Ihe graphic Jayoul app.ropriate for a. rural educational 

Setting? ^ . 

15. Ar^he^charts and graphs easy to read and upderstana? 

16. ^ l5o^the visual aids promote. better understanding of 
the product content? 

E- General Questions 
^ — 17. 

result from the use of this product? 
18. What are the "reviev/er's genef^l overall impressions 
of »the. product? . T , 



)1 Questions . . ' • 

What positive or negative side effects are likeiy^to 



The: Desig n of thte Exploratory Test t. . 
= • Q 



To 

read 3nd 



ansv^r the above questions the REP employed panelists to ■ 
.-review each product- The panel review apprpach. was selected 
"for Vo reasons. First; it. is a relatively efficient and inexpensive 
way to -collect data ^concerning needed revisions. Second, it provide? 
an opportunity to acquire information from multiple- sources. 

^ In the Exploratory Jests reviewer annotations and comments about 
the product as -welPas their questionnaire responses were summarized 
and provided- to the development team. The interactive procedures 
beWen developers and evaluators in these tests are discussed below. . 

•A preliminary meeting yvas held between developers and evaluators 
to specify the procedures for^e ejcploratory tests. "At tiiis meeting 
four reviewer populations were ideWified as appropriate participants 
for tile exploratory test. , . 

. 1 . Experts in "the product's substantive areas , . 

2/ Potential users of the»product ♦ ' 
3 : r<E.P r^lleagues who had deveIoped-»related materials 
4.) NWREL personnel witii expertise related to tiie product 

/ (optional) .y ' f ' ' 

Tv/o additional qualifications were established. First, some-ot 
tiie -reviewers had to be able to provide information abmjj the existence . 
of critical competitive pr'oducts. 'Second, no rSvlewer/was to have . 
been directly involved .in th^eyelppment of the product. - This last 
qualification did not exclud/those v/ho had previously been employed 
exclusfvely for 'the purpose of critiquing the pi'od'uct. • 

-Procedures were also outiined for .identifying and selecting- reviewers 
developing schedules', constructing instruments, conducting the test 
and reporting results. ' ■ ' 



The minutes of this preliminary meeting were prejDared in outline. 
J|rm and presented to the AdmSmstrative team/ a" three-man team composed 
oi the coordinators of management, develdpment, and evaluation units. 
The Administrative team &ien approved (wiih mrnor modifications} the 
procedures -flfr U^e conduct »f the exploratory tests- 

To Help insure selection of a representative and qualified group • 
of "reviewers, development team leaders, prepared a list of qualifications 
and suggested names, in each of the four categories, of peopte who 
might revFew their products. These 'lists were provided to the Admini- 
strative team vvho identified a pool of prospective reviewers meeting 
the specified qualiHcations and selected an eight to ten member review 
panel Ibr the product/ With one exception a separate review panel 
was refected for each product: due to tfte interrelate aqd complemen-. 
tary natOre of the process facilitator^ manual ^and training- guide, only 
one panel was. selected to review both {hese products. Lists of the 
review panelists for each pro'duct are presentej^ Jn Appendix A. 

Each panelist ^v/as asKed to^sign an^ agreement stating his or 
her willingness to serve as a revfewer until the completion of the explor- 
atory test. £kch externa) reviev/er received an initial stipend with 
the understanding that^ complete remunejration would await receipt of 
data, * " ' ' ^ 

Because of the development schedules and product Jength, it 
was agreed that each product could be divided and tested iii up to 
six distinct pieces. This agreement also permitted reviewers tg,examin.e 
portions .of- the product^ .sequentiaHy rather than to reviey/ the*complete_ 
product at one time. In accordance with this decision, the pevelop- 
mental Schedule of Product Pilces for Exploratory Jesting >vas developed. 
See Appendix B, Based on^ it^ an Exploratory Test ScViedule wa# also 
developed^ establishing^ as a minimum, eight working days for production, 
one calendar week for review, and six days for ^jailing each .product 
(three days to and three days from the reviev/ers) . The only* exception 
to this schedule was that two weeks were allowed for review of Piece 
1 of the training guide. Appendix C presents the Exploratory Test 
Schedule: , 

instrument Construction 

- ^ The exploratory test was designed to answ^er questions related 
to a product's objectives', content, format, and intended use. .Questions 
appropriate to all RFD products were identified in meetings with develop- 
ment team- leaders and the Visual Communications Unit, coordinator. 



A<3ditional!y, questions relating to specific products were identified 
by the ey^Iuator and development team leader for those products. 
A few questions specifically addressed to REP colleagues or NVVREL 
expert reviewers were also iden tiffed. 

Based woh these questions^ items v/ere constructed and a question- 
naire for each/RFD product was d^eloped. (See Appendix/D for a 
sample Exploratory Tes,t Questionnaire.) Before each test questionnaire 
wa^actu&lly used, it*v/as reviey/ed by members or representatives 
of the foll6wing groups:. , - <yy 

1. The Product Development Unit 

' - 2» The Research and Evaluation Unit ^ 
3. NWREUs Office of Research and Evaluation Services, ^ 
4.. . -The REP representative for the Protectipn of Human Subjects 
Conducting, the Exploratory Test 

At the initiation of each exploratory testj an orientation meeting 
v/as held (April 8 and 9, 1975), attended by the 'reviewers selected 
and product development and evaluation teams. ^ The purpose of the 
orientation was to ensure that the reviewers shared a common understand-- 
ing of their task. ^At the meeting each reviewer received: ' (1) a statement 
of RFD goals and olDjectives, (2) a statement of the product's- goals 
and objectives, :(3) an exploratory test schedQle' that ir^dica'ted the 
dates when product pieces would be mailed to reviewers, the review 
time for each, and the dates when review annotions and questionnaires 
shouliilDe returned. . * t ' / ' * 

The first review packet was presented to reviewers at the orienta- 
tion meeting, with successive packets maijefl (with minor modifipations) 
accTording to the ExploratoVy Test Schedule. Each review .packet consisted 
of a questionnaire and two copies of the product piece tone for the 
reviewer to keep for \eference during Veview 6f ^subsequent pieces 
and one to annotate an« return). Reviewers were ^sked 'to' annotate- 
the materials,' respond to the product questionnaire, and return both 
annotations and questionnaires for analysis. Follow-up procedures"^ 
wer^ developed to guard against late return of review materials. 

The dates of major exploratory test events are presented in Table 



2 _ , ' ' 

'•One reviewer was unable to attend the orientation meeting^. Another * 
reviewer having e?cpertise in the "same area was, thus, asked to assume 
the original reviewer's task. ^ ^ 



.S .TABLE 1 , 

^scHedule^Of major events FQR* 

EX^'J.ORATORY TESTS . 



Event 

' 1. ^ Specific Development Schedutes for RFD Products 

^and Procedures for Exploratory "Tests Approved * ^ 
by the Administrative Team - 

Qualifications for' Panelists Prepared by Develop- 
ment Team Leaders ' ^ ' . ' • 

3. Exploratory Test Review Panelists 'Selected l?y , 
the Administrative Team , 

.4. ^ Exploratory Test Schedule Finalized ^ 

Exploratory Test Qoestionnai res Prepared 

6." Review of Questionnaire Completed by 

Developers, Evaluators/^NWREL's Office^ of ' 
Research and''Evaluation Services, an^ 
the Party Responsible for Protection 6f Human S - 
Subjects ^ * , 

* ji • . . » , . . . 

.7. Orientation Meeting of Review Panelists for the 

A. PF Manual and Training Guide 

B. Sphoolv^Cuide, School Board ^uide/ ^ 
- SQG Notebook ; ^ - - ^ 

8. Informal Exploratory TesfflepoV:ts- Prep'aredi 
for the " ^ - 

A-i PF ManuaL^Bieces 1-6 ^ . 

- Bt Training Guide, Pieces, 1-3 

C. Schooi Guide, Pieces 1-4 
^ JD<""^Schoof Board Gui^.e, Pieces 1t4 

. - ' E. . SCG Notebook/ Pieces 1-4 

.9, ' Formal Exploratory Tefet Reports - . 

A. PF^ Manual , * , 

B. "graining Guide, ' . 

C. School Guide 
' D. School Board Quide - " 

i E. SCG Notebook 



' bate 



March 5^ 1975 

, March 12, 1975 

March .17, 1975 
March 26, '1975- 
March 27, 1975 



Mai^ch 28, 1975 

April 8, 1975 
,Aprtl-9, 1975 



, April 8 to 
j'une-30, ^^75 



July 1^ to^,^- 
October 1975 
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Cala Atialysis and Reporiirg 



Fcr each product pi^e, an informal report was prepared by 
e-aluators end provided to developers within a tv#o-week period foliowlng 
receipt cf data. Tliese informal reports consisted of four types of data^^ 

1 - Tabulation cf types of reviewer annota Jons in thr*^ categories. 

positive cx:n:ments, negative comments aix3 suggest ^ alterations. 

2. Summary cf frequently suggested alterati ns from annotations 

3. Frequer^cy tabulations for forced choice questionnaire items 
Sumjrary cf frequent comments to open-eiced questiornaire 
items 

Reviewer ccmjnents and annote^OTS of frequency three or greater 
•vere always included in the informal reports, along with comments 
and annotations that occured Tess frequently when these promised to 
be useful in improving the product. This decision v/as made so that 
the reports would be more concise and helpful to development team 
leaders *vho were revising tri? product- , 

After the preparation of each informal report the developer and eva'- 
uator met to discuss the report including any comments and implication i 
of the data- Following this meeting the developer and evaluator interacted 
whenever necessary to clarify information and issues from the repofls^ 

An agreem^ent^was reached by the development and evaluation 
^*nits that tne development team leader Vi^ould assume the responsibility 
for making decision; aboutproduct revisions- Hov/ever, these decisions 
;vere subject to approval of the development coordinator and program 
director- 

Developers used the following criteria to^etermine which test 
data would be used as the basis for revisions: 

* Consistency with RFD assumptions^ research base, and field 
experience base. 

* Strength and frequency of comment made by reviewers. 

* Importance or significance of comment. 

* Usefulness to the development of the prototype product. 

* Helpfulness with correcSng unintentional errors or omissions. 

* Contribution to internal and cross-product consistenc/- 

* Likelihood of developers completing changes v/ithin their timelines 
and resources. 



3 

For samples of these forms see pages V-2 t^V-6 and VIl-4 of the Exploratory 
Test Report, RFD Guide for Training Scho^-Community Process Facilitators . 

Rural Education Program^ Northwest Regional Educational Laboratory^ 
1975- g 
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For each product, a final exploratory test report was prepared 
. jointly by Jhe developn^nt and evaluation teams - This report identified 
the smjor exploratory test results anc recomizended revisions based en 
these results. ?*!ajor results were defined as product strengths and vicak- 
nesses identified by forty percent or irore of the reviev/ers. 

For the final report evaluators prepared chapters on the •Desigr 
of the Exploratory Ter*s" anj "Wajor Exploratory Tesc Results." The 
results from the c»n:p51ed in^rma! reports were included as appendices 
in the final report. 

Developers prepared a chapter on the *Ccnclusions and Recc^irnen- 
dations* which documented the most signiHcant revisions based on the 
niajor exploratory test results. In no cases were high frequency explora- 
tory test data (40 percent or more) ignored; if the data v/ere not used 
as the basis of revisions, rationale statements were prepared to document 
the omissions. A complete list of ali proposed revisions from the explora- 
tory test data^ REP colleague review^ cxinsultant advice^ literature search 
and the developers own insights v/as aiso included as an appendix in 
the Snal exploratory test report. 

Resuits of the Exploratory Tests 

This section presents the major exploratory test results from^ the 
Training Guide^ one of the five RFD products. Although these results 
are specific to one product we feel they* are fairly^ representative of the 
type and content of the results from the other RFD products. 

In presenting the results under each of the questionnaire sections 
we will indicate the quest.pns posed to reviewers, the reviewers' respon- 
ses, and the developers* necommendations for revision.^'.^ 



For Additional information regarding exploratory test results and developers' 
revisions fa&sed on -these results^ see Appendices Vi, VIL and Vlll in the 
Exploratory Test Report. RFD Guide for Training School-Community' Process 
Facilitators ! Rural Education Program, Northv/est Regional Educational ' 
Laboratory,. 1975 . 

5 

Developer's recommendations for revision v/ere prepared by Hans Johnson 
and Greg Druian of the J^rthv/est Regional Educational Laboratory. 

10. 



A- Questions posed , Ini the area cf goals and objectives^ 
revSev/ers were asked 

t 

* if they perceived discrepancies between the product's 
goals and objectives and the particular section being * 
revieweS 

* if the goals and objectives v/ere stated clearly enough 

* if any should be eliminated^ addled^ or revised 

(See questions 1-^^ of the questionnaire. Appendix D.) 

8- Revievyers* responses , ' The reviewers indicated that the 
goals and objectives were generally comprehensive and 
well statedv 

C- Developer's recommendation - No revisions in goals or 
objectives are planned for this product. 

Content - , * * 

A. Questions posed - Revieivers were asked 

* to assess the scope and <^etaiI-of the product, paying 
particular attention to the guidelines for organizing 
and conducting training sessions 

* if materials need to be added to or eliminated from 
the product 

% 

* if th^ product was appropriate for a rural audience . 

*^ if the product content matched its goals and objectives 

*- 

^ if it was similar to other. materials v/ith which the 
reviev/gr v/as familiar ^ ^ 

(See questions 5-11 and 25-30 of the questionnaire.) 



B . " Reviev^ers* respcnses Four revie/*'ers felt that additicnal 
detaji vjBs needed in Piece 3. They felt that tt5e*guide 
presOTted enough informa^cn for organizing and conducting 
training sessions and for selecting appropriate trainif^ 
activities for the sessions. - 1 . 

Reviewers also indicated that there were il3ateria!s which 
should be added to or eliminated the product. The 
following are son;e of Hie changes which wfere suggested 
for a given product piece. . ' 



Piece 1: 

* . * - 

* Reviewers felt that trainers would need a workshop to teach thejn 

iheinrol^s and responsibilities. They also indicated thaL procedures 

should be specified fof periMic^assessment, quality'controi^ 

and updating of trainers in critical professional areas — kmwledge^ 

skills, and atti&ides.^ For Jhe first piece, revie\'7crs also indicated 

that inore examples v/ere necessary. 

Piece 2: 

The situation activities^ in this piece were seen as very helpful. 
They v/ere useful, relevant, and likely to achieve the goals set 
for-Oiem. . - 

Piece 3: 

Reviewers felt that repetition of the core activities for each phase 
^ was-ijnnecessary and that more information v/as. needed about 
buildmg PF teams. Activity F of the Orientatiofi phase was seen 
as v/eak anrf in need of improvement- } 

. '■ ■ ■ ■ 

The sample training activities were generally seen as useful, relevant, 
and likely to achieve their goals, though half the reviewers felt that some 
activities would falfshort of their goals. 

In summary, the content v/as viev/ed as appropriate for use in a 
rural edjjcational setting. The materials were generally easy to read and 
^understand, and the content was well matched with stated goals and 
objectives. In addition, few critical compgttive products vyere identified 
tiiat could be used in place. of this product or its parts.. 
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C. Develcper^s Recommendaticn . Several contenl revisions^ mostly 
in the S3n:ple training activities^ are planned cn the basis of 
test results. * / 

Jn &e puide itself^ examples will be provided to illustrate ideas^ 
concepts^ and abstractions. These v/ill occur as narrative examples 
in the text and also as sample forms and guidelines in tiie various 
rescSurce sections: . 

One misconception shared by several reviewers was that ^e 
guide would also be used to train PF trainers. Since this is not the, 
case, several suggestions for procedures, rtools, and resources that 
relate to'PF trainer training v/ill not be Incorporated in the guide. 
However, Iflie introduction will be rewritten to clarify the scope of 
the guide.' _ . * - ' ^ • 

In the sample training activities, several samples will be added 
to focus on PF %vork in problem-solving applications and group facilita- 
tion with the school staff, school leaders, and the school board. ^ 

The core activities will be presented only once, early in the * 
sample training activities, rather than attached to each training phase. 
Also, one page of Core Activity F v/as misplaced, leadiil§[ reviewers 
to ojmment that enough information v*as not provided on PF team building, 
the subject of the activity. This page will be restored in the prototype 
version, and some additional information on PF team building will also 
be added. ^ 

Additionally, the goals of all activities will be revised to focus 
on realistic PF outcomes rather than trainer outcomes, so that the activities 
can -reliably be expected to accomplish the goals. 

III. Intended Uses 

A. Questions posed. Revievi?ers were asked to consider 

^ ' * 

* hov/ v/ell the product piece accounted for the important 
characteristics of its intended users 

* if it v/as necessary to include the product in the 
RFD Strategy 

* the adequacy of the resource materials included or 
referenced in the product 

13 
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. * - the adequacy cf the information ccncernang additional 
ristAirces - 

(See ques^cns 12-15 cf the questionnaire.) 

B. Reviewers' responses . Reviev/ers were ambivalent \p their 
remarJjs concerning the characteristics ci intended user 
populations. Vn Pieces 1 and 3^ reviewers fe!t that the 
characteristics cf intended users needed more attention. 
However^ they -felt that Piece 2 adequately considered the 
in^portant^characteristics cf its intended users - 

All reviev/ei3 considered the product an essential element 
in the RFO-Strategy. They felt that the resource materials 
were adequate and that there was enough information io 
assist users in locating additional materials. 

C. Developers' recommendation. No single revision is planned 
to address the issue of the product's match viritii its intended 
user population. Rather, the entire narrative and all training 
activities v/ill be revised with audience diaracteri sties in 
mind. 

Although resources appeared to be adequate and reviewers 
found them helpful, they suggested other resources, which 
will be included, in the prototyp^e product. In additic^, 
more frequent cross-referencing to resources in the guide 
and the PF manual wilf be added. 

Also cn the basis- 43f reviewer* cgmment, a more o>mprehensive 
discussion of adajDting training activities will be included 
in the guide, and all time estimates on sample training 
. ac.tivities will be reviewed and, if necessary, modified* . 

Format and Graphics 

A. Questions posed . In regard to ihe product's format and 

graphics^ reviewers v/ere asked to pay particular attention 
to <^ 

* * • 

* the sequence of material in the product piece 

* the table of contents, headlines, and titles 

14 
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« * Ihe usability of the charts and graphics 

*. the ger^eral graphic layout, paper o^lCr^ drawings^ 
symbols^ and ether visual aids 

(See questions 17-24 of the questionnaire.) 

8. Revie*i>vers* responses , Reyiev/ers generally agreed that 
• the material was appropi^tely sequenced within the product 
• ^and that the teble of <X3ntents^ headlines^ and titles were 
, ' satisfactory. TTjey noted^ however^ that the indexing ^stem 

for resources should improved to allow ^sier cross- 
referencing- The charts and graphs within the product 
were generally easy to read and understand^ but half of 
the reviewers indicated t^at the charls*4n Piece 1 should 
be eniacged. Reviewers also indicated that the format 
oflhe hol^^k was satisfactory. 

In Pieces 1 and 2* at least' 50 percent of the reviewers felt 
Qial the graphic layout v/as appropriate for use in a rural 
educational setting. Reviewers indicated for Piece 3 that 
th^ visuals should be mor^ artistic and crrative. For Pieces 
1*and 2, half of the Reviewers indicated that changes v/ere 
needed in the graphic layout. While they thought that 
the visual aids promoted understanding of the material^ ^ 
two-thirds of them noted that the drawings in Piece 1 should 
relate more closely to the conteni^material , 

C. ^ Developer's Recommendation . The complexity of the labeling 
system for^e sample training activities will be greatly 
reduced ana Ihe system clearly described on a sample page. 
This revision and the inclusion of onty one description 
of core activities will result in grater ^se in cross-referencing'^ 
the activities. ' ' ^ 

^ . Where charts aVe used^^ the type face will be at least 6 
point for legibility^ and the layout will be simplified. 

The guide v/ill be illustrated v/ith a continuous graphic 
' design^ flowing throughout the test> to add visual interest^ 
^ * and to break up long narrative passages. 
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V. Sufgmary Concerns 

A- Questions ^posed, Revie*vers were asked 

* to indicate the positive and negative aspects of the 
product . 

* ^whether the changes they suggested were relatively 
^rtiinor or ^ . • 

if they were_ major ones requiring sicfnificai^t revision^ 

. * if they felt that develcpnient oPthe prociuct shoujd 
cease 

(See questions 32^35^oT the questionnairej* 

8- Reviev/ers' responses . One-half of the reviev^rers felt that 
only minor changes in the product were needed^ while 
four reviewers suggested that major changes were needed 
for the product: None of th^ reviewers recommendecl that 
development of the produc* should cease. 
- \, ^ . 

C- Developers' recommendation. Although several reviewers 
'suggested major revisions to the guide^ these revisions- 
were generally unique for a given reviewer. Most suggestions 
have been addressed in preparing the above recommendations 
for revision. 

Conclusion \ * , 

^his paper has attempted to show that xievelopers and evaluatdrs*' 
can work together quite effectively in the formative evaluation process- 
, Such interaction should lead both to the development of high quality 
educational products and to more cooperative relationships bftween 
developers and evaluators. The exploratory test plan and procedures 
presented in this paper may be of sfcme value for other developers 
and e valuators v/ho are engaged in a similar enterprise- 
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. APRhnDiX A. 
EXPLORATORY TEST PANELS 



L K^lanual/Trafning Panel 



Reviewer 



1: 



2. 



3. 



-V/iiiiam Lassey 
Department 9f Rural Sociology 
Washington S'fete University 
Pullmanv V/ashiicigton 

Charles DeRitter\ 
Regional Training Officer 
U. S. Forest Service * * * 

Mark Milliman 
School Board Chairman and 
Instructor, Division of 
Continuing Education 
Oregon State^Systenf of . 
Higher Education 

David Curry A 
Secondary Ec^ucation and 
.Administration Specialist 
Oregon State Department of . 
Education \ • ' 



\ 



Wiltiam Ferguson 
English Consultant \ \ 
Office of the Superintendi^^ 
of Instruction * 
Helena, Montana 



7. 

t 



8- 
9. 



10. ' 



Martha Harris ^ 

Curriculum Specialist 

Eugene Public Schodli?^ Oregon 

Jane Arends-^ 
--Center for Education Pdlicy 
and Management 
University of Oregon ' 
formerly REP staff member 

Warren Adams ^ 

REP staff -member 
V ' - 

Ruth Emory ^ 
Improving Teaching Compentencies 
PjTogram 

Northwest Regional Educational 
Laboratory . 

Rene 'Pino 

Imfjroving Teaching Competencies* 
Program 

Northwest Regional Educational 
Laboratory 
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School Ciide Panel* 
Reviev/er 

« 

, 1 , Margaret Nelson 

Intermediate Education District 
Washington xCounty:^ Oregon 

2. Mark MilHman 

- School Board Chairman anS 
Instructor, Division of 
^ 'Continuing Education 

Oregon State System of 
Iflgher .Education 

m 

3. Richard Withy combe % 
Portland Public Schools/ 
Oregon ^tate University 

^. Dart-ell Clukey 

Oregon^ Episcopal Schools* >^ * 

5. Marilyn Curry ^ 
former school teacher 
Oregon Pu^blic Schools 

Jtr * m W 

6. ^ Garry Fendell 

Principal of Zillah High School 
ZMIa, Washington 

7. Bertha^ Mansker 
School Tfeach^r / ^ 



. .Staff Member' 

9. Samellyn' Wood ^ ^ • ^ 
Freelance Develpper and 




8- ' Ray Jong6ward, 



Evalualor 
formerly REP Staff Member 
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!V. School Community Group Notebook 
, Reviewer 

1. Lawrence Horyna 
College of Education 
University of Oregon 

2. A- D. Luke 

State Department of Education 
Boise, Idaho 

Richard VVithycombe 
Portland Public Schools/ 
Oregon State University 

» 

Gilbert Anzaldua 
Director of Compensatory 
Education Programs 
Oregon State Department^bf 
Education* 




5.. William Monroe 
S^ihool teagher 
Sheridan, Oregon 

6, Pat Tift - . 
former member of School' 
Communi ty <j roup 
Brewster, Washington" 

7. " Jane Arends 

Center for Educational- Policy 
and Management / J 
University of Or^on 
* formerly REP staff- member 

^ 8* Keats Carman 

REP staff-member 

„ ■ -.- 
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HL School Board Guide Panel 
Reviewer 

.• - \ 

^ 1 . Clyde Brown 

School Superintend^t 
Brewster^ Washington 



2. Ray Talbert 
Educational Coordinates 

' Northwes.t 
Eugene^ Oregon 

3. ^ Max Abbott 

^ Director of.ihe Center for the 
Advanced Study- of Educational 
Administration 
University of Oregon 

4. Alta Fosfaack 

School Superintendent 
Carlton, Oregon 

5- .Mark -VHIIiman 

^ -SchoohBoard Chairman and 

Instructor^ Division of 
^ Continuing. Education 

Oregon State System of 

Higher Education 

t 
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6. Ragnor Anderson 
Sdhool Board Member* 

- Co I ton, Oregon ^ 

7. Ray Haag 

As5istan5 Superintendent ; 
' Intermediate JEducation District 
Washington County, Oregon 
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Jane Arends 

Center for Educatii::inal Policy 
and Management 
University of Oregon n 
formerly REP Staff member 

Roger Bishop 

Freelance curriculum developer 
and consultant 

formerly REP steff member * 
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APPENDIX 8 
-DEVELOPMENT SCHEDULE 
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Expioratoiy Test Schedule 

Assunpticns 

1. A packet sent to reviewers vlli include the folicr-;ing 

o 

a, A copji of the goals and objectives of the ^ 
product 

Two copies of the product piece 
c^ A questionnaire for the product 
d. Any integral resource saaterial referenced in 



2-. 
3. 



the product piece; 

Eight: -z^-orking^ days are allotted for editing and 
production tine- 

One calendar n^eek is allotted for revie;^ of each 
product piece ^ 

f 

0 

Exceptions r 

a. Piece 1 of the Training Guide and Materials is 
allotted t:;o calendar veeks for xevxew tine* 

Piece 2 of the Process Facilitator Manual wiil 
be saailed to revie'.^ers during the tisie allotted 
for review of Piece 1 of the Training Guide and 
Materials* Due to this* these t,\<o pieces, should 
^be revieared concurrently during that time. 



4* Six calendar days have been a3.1otted for jnailing tine 
to and from reviewers. 
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APPENDIX D 



Dear Reviev^er: ' ^ - . ' 

Enclosed you vill find ^ 

of the RFD product: . 

Your annotations and review of this product pifece will provide 
inportant information for re\''ision of the product. 

This packet includes: ^ 

1. Tko copies of the product piece 

2. A list of the goals and objectives of this product ^ 
piece - ^ 

3. A questionnaire about the product piece 

4. Any integral resource material referenced in the 
product piece - \ . 

You should have received this packet T)y ^ We request 

that you return your annotations and the questionnaire in the 
self-addressed stamped envelope by If problems 

arise, please contact us by phone (503/224-3650) - 

As you begin your reviiew it wo^d be helpful if you would 
read (reread) the following: - , " 

^ 1. The goals and objectives of the complete product 

2. The goals and objectives of the product piece 

5. The questionnaire for the product piece 

» » * 

As you review the materials please annotate one copy of* the 
product piece as completely as po^ssible. Please write your, 
comments adjacent to the appropriate section. In particular 
we would like you to point out strengths and weaknesses in. 
relation to: ^ 

1. - Theoretical/ underpinnings * 

2. Organization * . " ' 

t 22 \ 



5. Relevance 

4. Clarity • ' 

- • • \ 

5. Practical uses - ^ 

6. Illustration/graphics * "^^^ 

7. Exainpies 

8. Continuity \ 

' - 0 

. 9. language level . •* • 

Feel free to use the space on fhe back of the sheets, .should 
you desire, and to he as pointed and expressive as you wish! 

The second copy of the product piece is for you to Tceep so 
that you can gain an increasing sense of the-total product 
as individual pieces are sent to you. 

After annotating the one copy of tHe materials please fill out 
the questionnaire for the product piece. Th^e questionnaire 
will focus on* questions regarding, the objectives, content/ 
production and intended uses, of the product piece. We feel 
that a combination of your open-ended comments on the product 
piece itself, and the more focused data from the questionnaire 
will give us a 'balanced and extensive set of suggestions from 
which we thrill write new product drafts* 

When you have -completed your review pleas^ mail:' 

1. The annotated product piece^ 

2. The completed questionnaire for the/ product piece 

Any comments on this review procedure 

Thank you for your input and help. in the development of a 
potentially useful educational product* ' 

Sincerely/ ' ^ ' \ 



Team Leader 

Rural Education Program^ 

3H 
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Rural Education Program, iNWREL 
. Product Test Questionnaire* 



Product ^ ^'iece, _ 

Date " Reviei^er's signature 



% 

INSTRUCTIONS . 



The^ purpose o£ this questionnaire is to obtain 
your opinion on several different aspects of 
the product in a "way- which , we can summarize 
relatively easily and then use in revisinjg the 
product- Your responses will be used by REP 
staff "as they revise the product''. 

• . - ' 

Please look through the entire questionnaire before responding. Most 
of the questions ask you to express your opinion on some aspect of the 
product by checking (X) a "yes" or "no" category. In some cases you 
will be asked to further explain your response. 

» * m " 

The questionnaire' accompanying each product piece will be -essentially 
the same each time. There mayjbe, however, a few question SvXoward - 
the end which deal with some .special concerns we, have about the par- 
ticular product piece' you are reviewing. 

If in your view a particular question is not applicable to the product 
piece you are reviewing, please write "NA" next to .the question number 
If you feal'you lack sufficient information to answer the question, 
write VLSI" beside the que'stion number. 

We feel your critical comments,, and responses to, the questions below, 
. in addition to those made on .the product piece, will, be very helpful 
to us in revising our products. 



If you need additional ^space for your comments for any 
question, please use the back side of the "page. 



*This questipnnaite is designed for use during the Exploratory 

Testing phase of the RFD Strategy. - * 
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A- Coals And Objectives Of The Product 

1. You received a set of the goals and objectives 
for the total RFD Product (in the Product 
Specification Document presented at the orien- 
tation meeting) and a set of the goals and 
objectives for this product piece (included in 
this current review packet) • Do you perceive 
aay important discrepancies between these two 
sets of objectives? (Check either "yes" or "no") YES NO 



If YES, explain the discrepancies :^ 



2. .Are the goals and objectives of the product 
piece stated clearly and with sufficient? 
detail so it is possible to determine success 
or failure of the product -in meeting its 
objectives? 



If-?K), explain tvhich goals and objectives 
should be stared more clearly: 



36 
40 



3. Are there goals or objectives which should be 
elininated, added, or revised? 



YES t:o 



If Y£S, list the goals or objectives under the 
appropriate category : 



To 3e Added To Bs Elininated to Be Revised 



4. Please nake any further conunents about the 
goals and objectives "of €3ie product piece • 
- here: " . , 
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B. Content Of The Product 

S. Does the con ten c possess sufficient scope and 
detail to serve the purposes and the needs of 
intended users? ' YES NO 



If NO, explain the deficiencies as specifically 
as possxole : 



6. Do yon feel that prospective users of this 
product would find the materials easy to read 
and understand (e.g.,. are the vocabulary and 
writing style appropriate for the prospective 
audiences)? 



If NO, explain the difficulties you perceive : 



7. Does the content of the product piece match 
up with its goals and objectives? 



If NO, explain the inconsistencies : 



7, 



ERLC 



38 

/ - 42 



S. Is there any material which should be added 
or eliminated in €his product piece? 

i 



YES 2iO 



1 



8. 



If YES, explain what aaterials should be 
added or eiininated- 



Material lo Be Added 



Material To Be Elininated 




9. Is the content o£ this product piece appro- 
priate -for a rural education setting? 



If HQ explain what is^inapprppriate : 



10. Do you know of any similar product(s) which 
' we could use in lieu of this product piece? 



If YES, what are those similar products? 



9- 
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11- Please p^ke any further ccnnents about the 
content of the product piece here: 

! 



^0 ^ 



Intended Uses Of The Predict 

12\. Does the product piece adequately consider 

the important characteristics o£ those people 
vith whoa it will be used? 

If ?%'Q, vhat characteristics are not adequately 
considered? - [ 



12. 



13. Based on your understanding o£ the XFD Strategy, 
vould you consider the use of this piece essen-"* 
tial to the RFD Strategy? 



13. 



If NO, Khy not? 



14. Do you think that the resource material ^ 

referenced in the product piece and sent to , 
you, is adequate to meet the needs or 
intended users? 



If NO, uhzt additional laatgrials do you view ! 

as essential resources ? ' - 

— : ' I 



14. 



^1 



i'5. Has enough information heen^ provided for 
intended users to identify and use addi- 
tional resources (e-g., people, agencies, 
literature, etc.) besides those included 
in the materials? 



y£S NO 



If NO, yhat suggestions do yoa haye for 
additional infornation? 



16. Please nake any further consents about the 
product's intended uses here: 



15. 
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D. Fomat And Graphics Of The Product 

17- is the naterial appropriately sequenced within 

this product piece? - YES 2fO 



If 2iO^ explain inappropriate sequences : 



18. Does the table of contents as shoKn in the 
product specifications provide a detailed 
^and easy reference to sections of this 
product piece? - ' " 



If NO, suggest vays to ipprove it : 



19. Are the charts' and graphs for this product 
piece easy to re^d and understand? 



If NO, specify what changes would improve 
their clarity: ! 
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17- 



18. 



19. 



/ 



I 



20. Are changes needed^ in the general graphic 

layoiit of tnis product piece, the paper- color, 
or drawings and symbols to nalce then appro- 
priate for a- rural education setting?- • YES XO' - 

, 20. 

■ If Y£S, what changes would you suggest? 



. 21- Do the visual aids of the product piece 

pronote better understanding of.the content 
material? • ' - 



If NO, what changes could iuprove under- 
standing of rhe content materlal? ~ 



22- Are .the headlines and titles satisfactory for 
helping to organize and reference the content 
sections? 



If NO, what' would improve the headlines and 
titles? 



21. 



22. 
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23- Does fornat of the product facilitate • 

saodifying', reaoving or adding naterials? YES NO 



4 

If NO, Khat changes should be nade? 



24. " Please make any further comments about the' 
product format and graphics here: 



US 
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23. 



Specific Concerns About The Process Facilitator Manuals 

25. ' Is enough infori^axion prov^ided in this product 
piece about the role of each of the following 
groups in the RFD Strategy? 



fij 


State Education Agency 


25. {Ij 


(2) 


^Regional Educational Agency 


(21 


(3) 


Local School AdministrSrtors 


(3) 


(4) 


Process Facilitators 


(4) 


(5) 


School Staff 
School Board 


(5) 


C6) 


(6) 


C7) 


School-Conmunity Group 


(7) 



If you gave a NO response to any of the above, 
what: additional information do you think is 
needed? I ^ 



26. ^s enough information provided in this product 
piece about the r,elationship of the process 
facilitator to each of the following groups? 



(1) 


State Education Agency 


26- CI) 


(2) 


Regional Educational iCgency 


(2) 


(3) 


Local School Administrators 


(3) 


(4) 


* 

School Staff 


(4.) 


(5) 


School Board 


(5) 


(6) 


School -Community Group 


(65 



If you gave a NO response to any of the above ^ 
' what additional, information do you think. is 
needed ? 



'27. Axe the self -assessnent tools, i£ any, 

included in tnis product piece appropfiate 
in nunber and kind? 



YES KO 



27. 



If XO, how could they be isaproved ? 



28.. Does the product piece -provide enough 

information ^about sources of support and 
resources needed to carry out process 
facilitator tasks? 



If NO I what additional inforination is needed? 



28, 



PLEASE GO TO ITEM 32 WHICH BEGINS THE SU^^MARY CONCERNS ABOUT 
THE PRODUCT * ^ 
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F. Sunoary Concerns About The Product ^ 

32. Does this material "turn you on" (i.e., if 

you Here not a revieiver, would you read this 
nateriaij? 



YES NO 



If NO, tchat ivould make it more interesting ? 



32. 



33". fljhat positive or negative side effects might 
you expect from the use of this product piece? 



Positive 



Negative 




ERIC 



34. What is your overal^l recommen^tion concerning this 
product piece (check one).? 

(a) 



(b) 



(c) 



I consider the changes I have suggested 
relatively minor and- would recommend that 
this product piece be included in the total- 
product pretty much as it is. ' ' 

I consider that some of the changes I have 
suggested are rather major and would recommend 
tjiat this piece be significantly revised (as 
indicated in my preceding comments and/or on 
the anno.tated copy) before inclus^ion in the 
total product. 

I recommend that the development of this 
entire product cease. * 

^ • ' us- 5.2 ^ - 

« 



Ob. 



Sumnarise below your general impressions of this product 
piece. If you selected alternative (b) in question 34 
above, please specify the najor changes you would suggest 
for the product piece. * 
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Specific Questions For ^^*fREL Enployees 



36. Is this product consistent with the values 
and standards of the Northwest Regional 

Educational Laboratory? YES XO 



36, 



If XO, what changes are needed to neet these 

Values and standards? 
' # 



37. (To be answered only by REP staff) 

Ijs this product consistent and compatible 
with other products in regard to the 
following criteria?' ^ 

(1) Values 37. (1) 

(2) Style ' ' (2) 

C3) Termlnodogy' - (3)^ 

(4) Format ' (4) 

If you gave a NO response to any of the above, 
what ch^ges woltld make the product, more 
' compatible with other RFD products ?^ ~ 
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